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Abstract The applicability of model checking is hindered by the state 
space explosion problem in combination with limited amounts of main 
memory. To extend its reach, the large available capacities of secondary 
storage such as hard disks can be exploited. Due to the specific per¬ 
formance characteristics of secondary storage technologies, specialised 
algorithms are required. In this paper, we present a technique to use sec¬ 
ondary storage for probabilistic model checking of Markov decision pro¬ 
cesses. It combines state space exploration based on partitioning with a 
block-iterative variant of value iteration over the same partitions for the 
analysis of probabilistic reachability and expected-reward properties. A 
sparse matrix-like representation is used to store partitions on secondary 
storage in a compact format. All file accesses are sequential, and com¬ 
pression can be used without affecting runtime. The technique has been 
implemented within the Modest Toolset. We evaluate its performance 
on several benchmark models of up to 3.5 billion states. In the analysis 
of time-bounded properties on real-time models, our method neutralises 
the state space explosion induced by the time bound in its entirety. 


1 Introduction 

Model checking [3] is a formal verification technique to ensure that a given model 
of the states and behaviours of a safety- or performance-critical system satisfies 
a set of requirements. We are interested in models that consider nondeterminism 
as well as quantitative aspects of systems in terms of time and probabilities. Such 
models can be represented as Markov decision processes (MDP [31j) and verihed 
with probabilistic model checking. However, the applicability of model checking 
is limited by the state space explosion problem: The number of states of a model 
grows exponentially in the number of variables and parallel components, yet they 
have to be represented in limited computer memory in some form. Probabilistic 
model checking is particularly affected due to its additional numerical complex¬ 
ity. Several techniques are available to stretch its limits: For example, symbolic 
probabilistic model checking [2], implemented in the Prism tool [221, uses vari¬ 
ants of binary decision diagrams (HDD) to compactly represent the state spaces 
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of well-structured models in memory at the cost of verification runtime. Par¬ 
tial order [3] and confluence reduction [3S] deliver smaller-but-equivalent state 
spaces and work particularly well for highly symmetric models. When trading 
accuracy for tractability or efficiency is acceptable, abstraction and refinement 
techniques like CEGAR [53] can be applied. The common theme is that these 
approaches aim at reducing the state space or its representation such that it fits, 
in its entirety, into the main memory of the machine used for model checking. 
An alternative is to store this data on secondary storage such as hard disks or 
solid state drives and only load small parts of it into main memory when and 
as needed. This is attractive due to the vast difference in size between main 
memory and secondary storage: Typical workstations today possess in the order 
of 4-8 GB of main memory, but easily 1 TB or more of hard disk space. Moreover, 
with the advent of dynamically scalable cloud storage, virtually unlimited off-site 
secondary storage has become easily accessible. For conciseness, we from now on 
refer to main memory as memory and to any kind of secondary storage as disk. 

In this paper, we present a method and tool implementation for disk-based 
probabilistic model checking of MDP. Any such approach must solve two tasks: 
State space exploration, the generation and storage on disk of a representation 
of the reachable part of the state space, and the disk-based analysis to verify the 
given properties of interest based on this representation. The core challenge is 
that the most common type of secondary storage, magnetic hard disks, exhibits 
extremely low random-access performance, yet standard memory-based methods 
for exploration and analysis access the state space in a practically random way. 


Previous work. Exploration is an implicit graph search problem, and a number 
of solutions that reduce the amount of random accesses during search have been 
proposed in the literature. These fall into three broad categories: (z) exploiting 
the layered structure of breadth-first search (BFS) by keeping only the current 
BFS layer in memory while delaying duplicate detection w.r.t. previous layers 
until the current one has been fully explored |12I38| : (ii) partitioning the state 
space according to some given or automatically computed partitioning function 
over the states and then loading only one partition into memory at a time in an 
iterative process jam; (Hi) treating memory purely as a cache for a disk-based 
search, but using clever hashing and hash partitioning techniques to reduce and 
sequentialise disk accesses m- Exploration can naturally be combined on-the- 
fly with checking for the reachability of error states, and methods to perform 
on-the-fly verification of liveness and LTL properties exist mm- 

The analysis of other logics, such as GTL model checking with satisfaction 
sets, and of other models, such as probabilistic model checking of MDP with value 
iteration, inherently require the entire state space for a dedicated analysis step 
following exploration. Previous work on disk-based probabilistic model checking 
considers purely stochastic models and focusses on the analysis phase: In ab¬ 
sence of nondeterminism, classical block-iterative methods |33j can be used with 
disk-based (sparse) matrix representations of Markov models. They proceed by 
loading into memory and analysing one matrix block at a time (plus those that 








it depends on) iteratively until the method has converged for all blocks. Im¬ 
plementations can be divided into matrix-out-of-core and complete out-of-core 
approaches m- In the former, the vector of state values being iteratively com¬ 
puted is still kept in memory in its entirety m- It is similar to how Prism 
uses BDD in its “hybrid” engine for the model only, while both model and 
values are represented symbolically in its “mtbdd” engine. The symbolic and 
disk-based approaches for Markov chains can be combined Further work 
on the disk-based analysis of purely stochastic models includes different imple¬ 
mentations that are both disk-based and parallelised or distributed coni. 

For the nondeterministic-probabilistic model of MDP that we are concerned 
with, the default scalable analysis algorithm used in model checking is value 
iteration, an iterative fixpoint method that updates the values of each state based 
on a function over the values of its immediate successors until all changes remain 
below a given error. We are aware of only one explicitly disk-based approach to 
value iteration, which associates the values to the transitions instead of the states 
and is based on sequentially traversing two files containing the transitions that 
have been externally sorted by source and target states in each iteration mi. 
However, external sorting is a costly operation, leading to high runtime. 

The correctness of value iteration depends neither on the order in which the 
updates are performed nor on how many updates a state receives in one iteration. 
This can be exploited to improve its performance by taking the graph structure of 
the underlying model into account to perform more updates for “relevant” states 
in a “good” order. One such technique is topological value iteration HD], based on 
a division of the MDP into strongly connected components. More generally, this 
means that value iteration can also be performed in a block-iterative manner. 


Our contribution. The technique for disk-based probabilistic model checking 
of MDP that we present in this paper is a complete out-of-core method. It 
combines the state space partitioning approach from disk-based search with a 
block-iterative variant of value iteration based on a very compact sparse matrix¬ 
like representation of the partitions on disk. In light of the disk space available, 
compactness seems at first sight to be a non-issue, but in fact is a crucial aspect 
due to the low throughput of hard disks compared to main memory. Based 
on a given partitioning function, our approach proceeds by first exploring the 
partitions of the state space using an explicit state representation while directly 
streaming the sparse matrix-like representation to disk. When exploration is 
completed, the stored partitions are analysed using a block-based variant of 
value iteration: It iterates in an outer loop over the partitions on disk, for each 
of which value iterations are performed in an inner loop until convergence. All 
read and write operations on the files we generate on disk are sequential. We 
can thus easily add compression, which in our experiments reduces the amount 
of disk space needed by a factor of up to 10 without affecting overall runtime. 

Our method has been implemented by extending the mcsta tool [TB] of the 
Modest Toolset The implementation currently supports the computa¬ 
tion of reachability probabilities and expected accumulated rewards. To the best 



of our knowledge, mcsta is at this point the only publicly available tool that 
provides disk-based verification of MDP. We have evaluated the approach and 
its implementation on five case studies. The largest model we consider has 3.5 bil¬ 
lion states. It can be explored and analysed in less than 8 hours using no more 
than 2 gb of memory and 30 GB disk space. Our technique is particularly efficient 
for the analysis of time-bounded properties on real-time extensions of MDP. In 
these cases, the overhead of using the disk is small and the enormous state space 
explosion caused by the time bounds can be neutralised in its entirety. 


2 Preliminaries 

The central formal model that we use are Markov decision processes: 

Definition 1. A probability distribution over a countable set f2 is a function 
/r G J7 ^ [0,1] such that = 1- support is support(/r) = {5651 

fi{s) > 0 }. We denote by Dist(I?) the set of all probability distributions over I?. 

Definition 2. A Markov decision process (MDP) is a triple {S,T,so) consisting 
of a countable set of states S, a transition function T G S' —)• for a 

countable subset i? C K. with T(s) countable for all s G S, and an initial state 
sq G S. A partitioning function for an MDP is a function / gS—>^{ l,...,fc} 
for some k gN with /(sq) = 1- 

For s G S, we call fi G T{s) a transition of s, and a pair b = {s', r) G support(p) a 
branch of fi, with s' being the target state of b and r being the associated reward 
value. MDP support both nondeterministic and probabilistic choices: A state 
can have multiple outgoing transitions, each of which leads into a probability 
distribution over pairs {s,r). A partitioning function / gS— >{!,...,n},nGN, 
divides the states of an MDP into partitions = { s G S | /(s) = *}• The 
partition graph is the directed graph (P, U) with nodes P={Pi|I<i<fc} and 
edges U = { {Pi, Pj) \ i^jA3sGPi,pG T{s), {s', r) G support(/r): s' G Pj }. 
It is forward-acyclic if there is no {Pi, Pj) G U with j < i. 

We are interested in the probability of reaching certain states in an MDP and 
in the expected reward accumulated when doing so. Since an MDP may contain 
nondeterministic choices, these values are only well-defined under a scheduler, 
which provides a recipe to resolve the nondeterminism. The verification questions 
are thus: Given a set of states F Q S, (i) what is the maximum/minimum prob¬ 
ability of eventually reaching a state in P over all possible schedulers {reachability 
probability), and (ii) what is the maximum/minimum expected accumulated re¬ 
ward once a state in F is reached for the first time over all possible schedulers 
{expected reward)? These quantities can be formally defined using the usual cyl¬ 
inder set construction for the paths of the MDP m- 

The computation of these quantities is typically done using value iteration, as 
shown in Algorithm[l| for maximum reachability probabilities. For the minimum 
case, we replace maximisation by minimisation in line [5j To compute expected 
rewards, a precomputation step is needed to determine those states from which 
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values :={si-^l|s€F}u{si-^0|se5'\-F} 

repeat 

error := 0 

foreach s ^ S \ F do 

Vnen, ■= max {E(./,.>6support(M) ' values{s') \ ^ G T(s) j 

if Vnew > 0 then error := max{ error, {v^ew — values{s)\/values{s) } 
values(s) := Vnew 


8 until error < e 

9 return values{ sq) 


Algorithm 1: Value iteration to compute max. reachability probabilities 


F is reachable with probability one and zero, respectively. This can be done with 
straightforward fixpoint algorithms over the graph structure of the MDP m- 
Using MDP directly to build models of complex systems is cumbersome. 
Instead, higher-level formalisms such as Prism’s guarded command language 
are used. They add to MDP variables that take values from finite domains. In 
an MDP with variables (VMDP), each transition is associated with a guard, a 
Boolean expression that disables the transition when it is false. The probabilities 
and reward values of the branches are given as real-valued arithmetic expressions. 
Every branch has an update that assigns new values (given as expressions) to 
the variables of the process. The semantics of a VMDP M is the MDP |MJ 
whose states are pairs (s, v) of a state s of M and a valuation v for the variables. 
Transitions out of s that are disabled according to v do not appear in |M], and 
the valuations of a branch’s targets are computed by applying the update of the 
branch to the valuation of the transition’s source state. A partitioning function / 
for a VMDP can be determined by an upper-bounded arithmetic expression e 
with values in N: f{{s,v)) = e(v) where e(v) is the evaluation of e in v. The 
reachability set F can likewise be characterised by a Boolean expression. 


Real-time extensions of MDP To model and analyse real-time systems, 
MDP can be extended with real-valued clock variables and state invariant ex¬ 
pressions as in timed automata (TA [3]), leading to the model of probabilistic 
timed automata (PTA [27]). A number of techniques are available to model- 
check PTA 121] , but only the digital clocks approach [SB] allows the computation 
of both reachability probabilities and expected rewards: Clocks are replaced by 
bounded integer variables, and self-loop transitions are added to increment them 
synchronously as long as the state invariant is satisfied. This turns the (finite) 
PTA into a (finite) VMDP. The conversion preserves reachability probabilities 
and expected reward values whenever all clock constraints in the PTA are closed 
and diagonal-free. However, the size of the final MDP is exponential in the num¬ 
ber of clock variables and the maximum constants that they are compared to. 

For timed models, we are also interested in time-bounded reachability: Ran¬ 
ging over all possible schedulers, what is the maximum/minimum probability of 
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Figure 1. In-memory representation of MDP for fast random access 


eventually reaching a state in F within at most t time units? These probabilities 
can be computed by adding a new clock variable x to the PTA that is never 
reset and computing the reachability probability for the set F' = {{s,v) | s £ 
F A v{x) < t} in the resulting digital clocks MDP m- 

A further extension of PTA are stochastic timed automata (STA [5]). They 
allow assignments of the form x := SAMPLE (D) to sample from (continuous) 
probability distributions D, e.g. exponential or normal distributions, in updates. 
This allows for stochastic delays, such as the exponentially-distributed sojourn 
times of continuous-time Markov chains, in addition to the nondeterministic 
delays of (P)TA. A first model checking technique for STA has recently been 
described [TH] and implemented within the mcsta tool of the Modest Tool- 
set [^. It works by abstracting assignments that use continuous distributions 
into finite-support probabilistic choices plus continuous nondeterminism, turning 
the STA into a PTA that can be analysed with e.g. the digital clocks technique. 

3 Disk-Based State Space Exploration with Partitioning 

In this section, we describe the partitioned state space exploration approach that 
we use in our disk-based analysis technique for MDP. We assume that the MDP 
to be explored is given in some compact description that can be interpreted as 
a VMDP, and a partitioning function / is given as an expression over its vari¬ 
ables. Disk-based exploration using partitioning has been the subject of previous 
work isns], so we focus on the novel aspect of generating a sparse matrix-like 
representation of the MDP on-the-fly during explicit-state exploration with low 
memory usage and in a compact format in a single file on disk. 


3.1 Representation of MDP in Memory and on Disk 

There are conceptually two ways to represent in memory an MDP that is the 
semantics of a VMDP: In an explicit-state manner, or in a sparse matrix-style 
representation. In the former, only the set of states of the MDP is kept, with 
each state stored as a vector {s,v = (ui, where s identifies the state in 

the original VMDP and Vi the value of its I-th variable. Given a state and the 
compact description of the VMDP, we can recompute transitions and branches 
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Figure 2. Inverse-sequential format to compactly represent MDP on disk 


at any time on-demand. The other alternative is to identify each of the n states 
of the MDP with a value in { 1,... ,n}, its index, and explicitly store the set 
of transitions belonging to a state index and the transitions’ branches. For each 
branch, its probability, its reward value, and the index of the target state need 
to be stored. This sparse matrix-style representation takes its name from the 
similar idea of storing a Markov chain as a sparse encoding of its probability 
matrix. All information about the inner structure of the states is discarded. 

FigurelH outlines the sparse matrix-style representation used by mcsta, which 
keeps three arrays to store the states, transitions and branches of a partition of 
the state space. For a state, “is target?” is true iff it is in the reachability set F 
that we consider. The target state of a branch is identified by its partition and 
its relative index within that partition. This format is more memory-efficient 
than an explicit-state representation when the model has many variables, and 
access to transitions and branches can be significantly faster because guards and 
other expressions in the model do not need to be evaluated on every access. 

The format of Figure [l| allows fast random access to all parts of the state 
space. However, when onH sequential access is required, an MDP can be stored 
more compactly. Figure shows the “inverse-sequential” format used by our 
technique to store state spaces on disk. States, transitions and branches are 
stored as a sequence of records, with the type of each record given by its first 
byte. Branches can be stored even more compactly by adding record types for 
common cases such as branches with probability 1. The key idea of the format 
is to first store all the branches of a transition before the transition record itself, 
and similarly store all the transitions (each preceded by its branches) of a state 
before the state record itself. In this way, we do not need to store the number of 
transitions and the index of the first transition for a state since its transitions are 
precisely those that appeared since the previous state record (and analogously 
for the branches of a transition). The random-access format of Figure [ij can 
be reconstructed from a single sequential read of a file in the inverse-sequential 
format, and the file can be created sequentially with one simultaneous sequential 
pass through the arrays of the random-access format in memory. 

3.2 Disk-Based Exploration using Partitioning 

Our disk-based exploration technique is given as Algorithm It is based on the 
approach of |5I16| . Files on disk are indicated by subscript o', when loaded into 
memory, the corresponding variable has subscript m- For each partition, we use 
BFS to discover new states (lines [12] to IMll with the following data in memory: 



















1 int count 1, g'ue'ueQ.append(so) 

2 repeat 


changed := false 
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// iterate over all partitions discovered so far 
for i := 1 to count do 

// Phase 1: update prelimina'ry target indices for cross transitions 
foreach j € successors'^ do array updates]^ := wpdaie5^.1oad() 
oldmat'rix\^ := matrix}:), matrix}:).clear{) // rename file 

foreach r G oldmatnx}) do // read records sequentially 

if r = (l,p, r,j, k) A k < 0 then 

matrix}:).append{(l,p,r, j, updatesi:^[—k — 1])) //update index 

else maina:D.append(r) 
unload updates\^ for all j € successors^ 
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// Phase 2: explore more states in breadth-first manner 
updates\y.clear {) 

queue queue\j^ := queue}^.load{), queue}^.clearQ, qlen^ 0 

indexed-set states\ji := states!).load() 

set done^ := states^ 

while queue]^.length > 0 do 

explicit-state s := q^aeae^-dequeueO 

if s ^ statesM then statesM-add(s), states!).append(s) 

updates !) . append ( states m .indexof (s) ) 

if s G done^ then continue else changed := true 

foreach t G s.transitions() do 

if ^t.guard{s.v) then continue 
foreach h G t.branches() do 

double p := 6.probability(s.t; ), r := 6.reward(s.u) 
if p = 0 then continue 
explicit-state s' := fe.target(s.r’) 

if f{s') = i then // local transition 

if s' ^ states^ then statesM-add(s'), siaiesi 3 .append(s') 
gMeMe]y[.enqueue(s') 

maina: 53 -append((l,p, r, i, statesM.indexof(s'))) 

else // cross transition 

j := f{s'), successors'.a.dd{j), count := max{ count, j} 
gMettep.append(s'), qlen^ — qlen'’ + 1 
maina:p.append((l,p, r, ji, — g/en-’)) // prelim, index < 0 

matrix}^. append{{2)) 

moina:n.append((3, s G F)) 

done' .add{s) 

unload queue'y^, statesM, done' 


39 while changed 


Algorithm 2: Partitioned disk-based exploration with sparse matrix creation 



— states'^: The set of states (explicit-state representation) of partition i is loaded 
into memory in its entirety when search begins for the partition (line 1141) . 
States are added in memory and appended on disk Hines [T51 and E51) . 

— queue^: The queue of states to explore in partition i. When a cross-transition 
is found during search in partition i, i.e. a branch leads to another partition 
j ^ i, then the target state is appended to queue^-^ on disk fline 1551) . For local 
transitions, the target state is appended to queue\j^ in memory fline 1291) . 

— done'": The in-memory set of fully explored states for the current iteration. 
When an iteration of search in partition i ends, states'^ is backed on disk, queue^ is 
empty, and done'’ is no longer needed, so we remove them from memory lline l38l) . 

During search, we simultaneously create the sparse matrix-like representation 
of the partitions on disk in files matrix}^ using the inverse-sequential format. The 
files are not loaded into memory. The records for new branches, transitions and 
states are appended to the file in lines [30l [34l [35l and [36l The main complication is 
the correct treatment of cross transitions: A branch record stores the partition j 
of its target state s' and the index of s' within that partition. However, we cannot 
determine this index without loading all of states^Y) memory, and even then, 
s' may not have been explored yet. To solve this problem, we instead use the 
index of s' in queue^ , which is easily determined (line I33|) . To distinguish such 
a preliminary index, which needs to be corrected later, from a local or already 
corrected one, we store it as a negative value Hine lMl) . 

The correction of these preliminary indices inside matrix\) happens at the 
beginning of an iteration for partition i Hines 15] to I 111) . The files updates^ for all 
successor partitions j are loaded into memory. These files have been created by 
the previous iteration for partition j in lines [T^] and [TH] and contain the correct 
indices for all states that were previously in queue^jy, at the same position. The 
preliminary queue-based indices in partition i can thus be corrected by a sequen¬ 
tial pass through its sparse matrix-like representation in file matrix'^, replacing 
all negative indices —k for partition j by the corrected value at [/c]. 

This is a random-access operation on the files updates^ ^ which is why they were 
loaded into memory beforehand, but a sequential operation on the file matrix^ ^ 
of which we thus only need to load into memory one record at a time. Observe 
that this correction process relies on the availability of updates^-Q successor 

partitions j. To assure this, we iterate over all partitions in a fixed order in lineHl 
instead of always moving to the partition with the longest queue as in [5I16| . 

To describe the memory usage and I/O complexity of this algorithm, let rimax 
denote the max. number of states, Smax the max. number of successor partitions 
(i.e. the max. outdegree of the partition graph), and Cmax the max. number 
of incoming cross edges, over all partitions. Then the correction of preliminary 
indices in phase 1 needs memory in 0(sniax ■ Cmax) for the updates^^A arrays 
and the exploration in phase 2 needs memory in 0{n-a^ay^ + Cmax) for states]^ 
and done' plus queue\A. Additionally, we need memory for the sets of integers 
successors', which we assume to be negligible compared to the other data items. 
A theoretical analysis of the I/O complexity [1] of a partitioning-based technique 
is problematic (and in fact absent from [5] and |I6| 1 due to the way multiple files 




// prepare value arrays on disk 


1 for i := 1 to eount do 


matrix]^ := matrix}^ .load{) 

for fe := 0 to matrixM -States. length — 1 do 

DaZweSo.append(matna;5^^.states[fc].istarget ? 1 : 0) 

unload matrix\ji 


6 while changed do 
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block-iterative value iteration 
// changed is initially false 


changed := false 
for i := count down to 1 do 

matrixlji ;= matrix^-loadQ, valuesM ■= valuesYj.loadO 
foreach j € successors^ do values\j^ := valuesf^doadf) 
repeat 

error ~ 0 

for fc := 0 to matnsM-States.length — 1 do 

if matnxM-States[fcj.istarget then continue 

Vnew := max... // as in Algorithm\^, but with valuesl^^/values^^ 
if Vnew > 0 then error :=...// eompute error as in Algorithm]^ 

[/cj .— Vnew 

if error > e then changed := true 
until error < e 

unload matrixM, values^ and the values^j^ for all j € successors'’ 


21 return values^lO] 


Algorithm 3: Partitioned value iteration for max. reachability probabilities 


are used e.g. when cross transitions are encountered: For the (unusual) case of 
very small Umax and very high Smax and Cmax, the disk accesses to append target 
states to different queues would be mostly random, but in practice (with low 
Smax and I/O buffering) they are almost purely sequential. A theoretical worst- 
case analysis would thus be too pessimistic to be useful. We consequently abstain 
from such an analysis, too, and rely on the experimental evaluation of Section!^. 

However, it is clear that the structure of the model w.r.t. the partitioning 
function will have a high impact on performance in general; in particular, a low 
number of cross edges is most desirable for the exploration algorithm presented 
here. Ideally, the partition graph is also forward-acyclic. In that case, two itera¬ 
tions of the outermost loop suffice: All states are explored in the first iteration, 
and the second only corrects the preliminary indices. 


4 Disk-Based Partitioned Value Iteration 

The result of the partitioned exploration presented in the previous section is a 
set of files in inverse-sequential format for the partitions of the state space. As 
mentioned in Section [J, value iteration can update the states in any order, as 
long as the maximum error for termination is computed in a way that takes 
all states into account. We can thus apply value iteration in a block-iterative 







manner to the partitions of the state space as shown in Algorithm]^. The vector 
of values for each partition is stored in a separate file on disk. In lines [T] to 0 
these files are created with the initial values based on whether a state is in the 
target set F. The actual value iterations are then performed in lines IH] to [201 
For each partition, we need to load the sparse matrix-style representation of this 
part of the MDP into memory in the random-access format of Figure [ll plus the 
values for the current partition (linejOl), those of its successors Qines fTUl) . 
The values of the successor partitions are needed to calculate the current state’s 
new value in line dSl in presence of cross transitions. Memory usage is thus in 
0(wmax + Smax ' ''T-max)) where rUniax Is the maximum over all partitions of the 
sum of the number of states, transitions and branches. The I/O complexity is in 
0(i-p-(scan(minax) + (smax + l)'scan(ni„ax))) where i is the number of iterations 
of the outermost loop starting in line [S] and p is the total number of partitions. 

In contrast to the exploration phase, the performance of this disk-based value 
iteration is not directly affected by the number of cross transitions. However, the 
number of successor partitions, i.e. Smaxi is crucial. An additional consideration 
is the way that values propagate through the partitions. The ideal case is again 
a forward-acyclic partition graph, for which a single iteration of the outermost 
loop (lineini) suffices since we iterate over the partitions in reverse order (line [5]). 

For expected rewards, we additionally need to precompute the sets of states 
that reach the target set with probability one and zero as mentioned in Section!^. 
The standard graph-based frxpoint algorithms used for this purpose m can be 
changed to work in a block-iterative manner in the same way as value iteration. 


5 Evaluation 

In this section, we investigate the behaviour of our disk-based probabilistic model 
checking approach and its implementation in mcsta on five models from the 
literature. Experiments were performed on an Intel Core i7-4650U system with 
8 GB of memory and a 2 tb USB 3.0 magnetic hard disk, running 64-bit Windows 
8.1 for mcsta and Ubuntu Linux 14.10 for Prism version 4.2.1. We used a timeout 
of 12 hours. Memory measurements refer to peak working/resident sets. Since 
mcsta (implemented in C#) and parts of Prism are garbage-collected, however, 
the reported memory usages may fluctuate and be higher than what is actually 
necessary to solve the task at hand. Our experiments show what the disk-based 
approach makes possible on standard workstation configurations today; by using 
compute servers with more memory, we can naturally scale to even larger models. 

Detailed performance results are shown in Table [ij. State space sizes are listed 
in millions of states, so the largest model has about 3.5 billion states. Columns 
“exp” and “chk” show the runtime of the exploration and analysis phases, respect¬ 
ively, in minutes. Columns “gb” list the peak memory usage over both phases in 
gigabytes. We show the performance of mcsta without using the disk to judge the 
overhead of partitioning and disk usage. Where possible, we also compare with 
Prism, which does not use the disk, but provides a semi-symbolic hybrid engine 
that uses BDD to compactly represent the states, transitions and branches while 


keeping the entire value vector (s) in memory during value iteration (limiting its 
scalability), and a fully symbolic MTBDD engine that also uses BDD for the value 
vector. The hybrid engine does not support expected rewards. 

Compression. As all hie accesses are sequential, we can use generic lossless com¬ 
pression to reduce disk accesses. Using the LZ4 algorithm [55], we achieved a 7x 
to lOx reduction in disk usage on our examples. We observed almost no change 
in runtime with compression enabled, so the extra CPU time is outweighed by 
reduced disk I/O. Compression thus lowers disk usage at no runtime costs. 

Partitioning functions. The actual performance of our approach depends on the 
structure of the model and its interplay with the partitioning function. Scalabil¬ 
ity hinges on the function’s ability to distribute the states such that the largest 
partition and the values of its successors ht into memory. The problem of auto¬ 
matically constructing a good partitioning function has largely been solved in 
prior work, and many techniques, like the ones described and referenced in |16j . 
are available, but they are not yet implemented in mcsta. For our evaluation, we 
thus use relatively simple manually specified partitioning functions. 

CSMA/CD: The MDP model of the IEEE 802.3 CSMA/CD protocol from the 
Prism benchmark suite. It was manually constructed from a PTA model using 
the digital clocks approach. It has parameters iV, the number of communicating 
nodes, and K, the maximum value of the backoff counter. The nodes count the 
number of collisions they encounter when trying to send a message. We parti¬ 
tion according to the sum of the collision counters of the nodes. The resulting 
partition graph is forward-acyclic since these counters are only incremented, and 
Smax = N. However, due to using the sum of several values for partitioning, the 
states are not evenly distributed over the partitions. 

We first report on the performance of computing the minimum probability of 
any node eventually delivering its message with fewer than K collisions (model 
CSMA/CD^^ in Table [H, with 1 X P indicating that one reachability probability 
is computed), and then on computing the max. and min. expected times until all 
nodes have delivered their message (model CSMA/CD 2 /g, where 2 x E indicates 
that we compute two expected-reward values). All MDP are only medium-sized. 
Our disk-based technique achieves performance comparable to the semi-symbolic 
approach here, which however does not support expected rewards. The fully 
symbolic approach has significantly higher runtimes for those properties. 

Randomised Consensus: The Prism benchmark of the randomised consensus 
protocol of N actors doing random walks bounded by K to reach a common 
decision. We partition according to the value of the shared counter variable. The 
resulting partition graph is strongly connected with Smax = 2. We use e = 0.02 
during value iteration (instead of the default e = 10“® as in the other examples). 
The MDP appear medium-sized in terms of states, but have about 5x as many 
transitions and 7x as many branches as states, so should be considered large. 


Table 1. Evaluation results (millions of states, minutes, and gigabytes of memory) 
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We check the two probabilistic reachability properties originally named “Ci” 
and “C 2 ”. The fully symbolic technique completes exploration and analysis much 
faster than our disk-based approach. This is because this model is a benchmark 
for value iteration, with values propagating in very small increments back-and- 
forth through all the states and thus partitions. Still, we observe that Umax is 
invariant under K, so our technique will be able to check this model for N = 8 
and any value of K without running out of memory—if given enough time. 


Wireless LAN: The Modest PTA model [H] of IEEE 802.11 WLAN, based 
on [2H]- So far, this protocol has only been analysed with reduced timing para¬ 
meters to contain state space explosion. We use the original values of the stand¬ 
ard for a 2 Mbps transmission rate instead, including the max. transmission 
time of 15717 n-s, with 1 n-s as one model time unit. Parameter K is the maximum 
value of the backoff counter. We partition according to the first station’s backoff 
counter, its control location, and its clock. The resulting partition graph has 
some cycles with Smax = 3. Exploration needs 5 iterations of the outermost loop 
of Algorithm 0 in all cases. We compute the maximum probability that either 
station’s backoff counter reaches K (model WLAN^p in Table[l|) as well as the 
maximum expected time until one station delivers its packet (WLAN^p). 


BRP: The Modest PTA model of the Bounded Retransmission Protocol (BRP) 
from m- Parameters are N, the number of data frames to be transmitted, MAX, 
the bound on the retries per frame, and TD, the maximum transmission delay. 
We fix MAX = 12. We partition by the number of the current data frame to ana¬ 
lyse the model’s six probabilistic reachability properties (BRP^^™). This leads 
to the ideal case of a forward-acyclic partition graph with Smax = 1- We also ana¬ 
lyse two time-bounded reachability properties (BRP^^^p) with deadline D and 
fixed TD = 32, partitioning additionally according to the values of the added 
global clock. This leads to Smax = 2. For the reachability probabilities. Prism’s 
MTBDD engine incorrectly reported probability zero in all cases. Our approach 
benefits hugely from having to perform far fewer total value iterations per state 
due to the favourable partitioning. In the reachability probabilities case, rimax is 
invariant under N, so we can scale N arbitrarily without running out of memory. 


File Server: The STA file server model from [TB]. C is the capacity of the 
request buffer. We compute the maximum and the minimum probability of a 
buffer overflow within time bound D. We cannot compare with Prism because 
some features necessary to support STA cannot currently be translated into its 
input language from Modest. Using our disk-based technique permits a finer 
abstraction for continuous probability distributions than before (p = 0.01 instead 
of 0.05). We partition according to the values of the global clock introduced to 
check the time bounds. This leads to the ideal case of an acyclic partition graph 
with Smax = 1- The state space and number of partitions grow linearly in the time 
bound while Umax remains invariant. We can thus check time-bounded properties 


for any large bound without exceeding the available memory, at a linear increase 
in runtime. This solves a major problem in STA model checking. 

6 Conclusion 

We have shown that the state space partitioning approach to using secondary 
storage for model checking combines well with analysis techniques built on graph 
fixpoint algorithms. We have used the example of MDP models and value itera¬ 
tion, but the same scheme is applicable to other techniques, too. In particular, 
the precomputation step for expected-reward properties is very close to what 
is needed for CTL model checking. Our technique is implemented in the mcsta 
tool of the Modest Toolset, available at www.modestchecker.net. In our 
evaluation, we observed that it significantly extends the reach of probabilistic 
model checking. It appears complementary to the symbolic approach: On the 
model where our technique struggles. Prism performs well, and where Prism 
runs into memory or time limitations, our technique appears to work well. In 
particular, our approach appears to work better for expected-reward properties, 
and we have been able to defuse the crippling state space explosion caused by 
the deadlines of time-bounded reachability properties in PTA and STA models. 
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